BigDFT build tree offers the possibility to have non-regression tests. When the user (or the devloper) types make check in the distribution, a number of tests are passed. Then a the fldiff_yaml.py script is checking that the logfiles of each run coincides with the reference results that are provided for each test. Ideally, each functionality of the code should have its own test, to ensure that further developments do not break it. It is therefore responsibility of the developer to provide new non-regression tests for each of the functionality. The optimal test should be

Reasonably fast (roughly less than 1 min on a monocore run)
Reasonably exhaustive (if the test passes this means that the functionality is indeed preserved - no false positive)

Clearly, these two requirements contradict each other. The art of the developer is to find a good compromise to reassure: 1) the other developers not to slow down the make check execution, and 2) himself, to be sure that new developments still would preserve the functionality.

Having said that, imagine that we have a set of files that we would like to use for non regression. There are different steps to include them in the test farm of BigDFT.

As an example, we show the insertion of the test for the molecular dynamics in BigDFT.

Prepare the directory

The tests should go in the tests directory (rather unexpectedly… ;)). The procedure described here is valid for four subdirectories: cubic, linear, overDFT and tutorials. All the other directories have special Makefiles associated and have a less easy way to automatize the test procedure. As our examples are related to MD, we put our tests in the overDFT directory. We create the directory MD:

cd tests/overDFT

bzr mkdir MD

Once this is done, all the test files inside this directory should correspond to a particular naming scheme. For example, a test named foo have to be associated to two reference files: named MD_foo.out.ref, containing the standard output (which essentially only contains one line with the name of the logfile), and MD_foo.out.ref.yaml which contains the lofgile associated to the test foo. Clearly, also the input files foo.yaml and foo.xyz (or any other supported format have to be present, as if it was a normal running directory. THe presence of the two reference files will cause the check procedure to run

$run_parallel ``\ \ ``/src/bigdft foo -l yes

where run_parallel is the environment variable containing the desired command to run tests. These tests will be ran in the directory tests/overDFT/MD-test/ of the build tree, to preserve untouched the input files and the references. The logfile of the run will be written in the file log-foo.yaml of the same directory.

Let us imagine we would like to insert two tests, with codename LJ-NVE and LJ-NVT. According to the above instruction, we therefore have to add the following files:

> ls MD/

> LJ-NVE.xyz  LJ-NVE.yaml LJ-NVT.xyz LJ-NVT.yaml  MD_LJ-NVE.out.ref  MD_LJ-NVE.out.ref.yaml  MD_LJ-NVT.out.ref  MD_LJ-NVT.out.ref.yaml  psppar.LJ

The directory might contain also extra files, that are needed for the run, in this case psppar.LJ. We should not froget to also add these files to the development branch:

> bzr add tests/overDFT/MD
adding tests/overDFT/MD/LJ-NVE.xyz
adding tests/overDFT/MD/LJ-NVE.yaml
adding tests/overDFT/MD/LJ-NVT.xyz
adding tests/overDFT/MD/LJ-NVT.yaml
adding tests/overDFT/MD/MD_LJ-NVE.out.ref
adding tests/overDFT/MD/MD_LJ-NVE.out.ref.yaml
adding tests/overDFT/MD/MD_LJ-NVT.out.ref
adding tests/overDFT/MD/MD_LJ-NVT.out.ref.yaml

Insert the test in the `make check` command

Even if the directory is ready, the test is still not active. To activate it in the make check procedure, the developer has to tell the build system that MD is a valid test directory. To do that, it is enough to insert the name of the directory (MD in our example) in the Makefile.am file of the tests/overDFT directory:

# Give here the list of existing tests
SHORT_TESTDIRS = FF-LENOSKY \
    FF-LJ \
    H2O-slab \
    H2-freq \
    MD #<<<< add it here...

[...]

LONG_TESTDIRS = $(SHORT_TESTDIRS) \
    C6H6-freq \
    GEOPT-all \
    GEOPT-BFGS \
    GEOPT-LBFGS \
        [...] #<<< ...or here

As it can be seen, the MD directory can be added either in the SHORT_TESTDIRS or in the LONG_TESTDIRS variable, but not in the two. Whether to choose one or another is a matter of the importance/robustness of the test: the tests in SHORT_TESTDIRS are ran by anybody typing make check on the released version whereas the others are only run by developers and expert users. Therefore a test in SHORT_TESTDIRS shoudl be.. short, and also should fail only when there are major instabilities in the code.

Running the tests and prepare tolerances

Even though an output file is prepared, we cannot expect the output of the test to be bitwise identical to the provided output. It is therefore responsibility of the ‘test forger’ to provide a set of tolerances for any of the fields of the output file (which for this reason is in markup language). This require some running of the tests to have an idea of the correct tolerances. To run the test the first time, let us go to the build tree (not in the source tree and execute a check for the MD test:

> cd ``\ \ ``/tests/overDFT

> make MD.check

This last command will, in the order:

Create the test directory MD-test in the build tree.
Copy the corresponding input files in such a directory.
Run the bigdft executable in the folder, with the command

$run_parallel ``\ \ ``/src/bigdft ``\ \ `` -l yes

where is each of the test runs (NVE and NVT in the examples defined above).

Once the test is finished, the fldiff scripts are ran to compare the logfiles of the output witrh the given references.

As all the log of the run are written in markup language, the different fields of the logfile might be compared with different tolerances. Such information is stored in the tests/tols-BigDFT.yaml file. Such file might be modified to adapt the tolerance of each field to the nature of the run.

Prepare the directory

Insert the test in the make check command

Running the tests and prepare tolerances

Insert the test in the `make check` command